In Norway, self-completion web questionnaires are dominant in all data collections from municipalities and in business surveys, and is about to become more common in social surveys as well.
Data collections from municipalities have been carried using electronic self completion questionnaires as the only collection mode since 2001. A little bit less than 50 questionnaires are used. From 2004 all business surveys, about 150 all together, have also been available on Internet. Until recently Internet has been offered together with a paper version that is enclosed with the invitation letter. But nowadays it is more and more common that paper versions only are given to those who call us and order it.
The content of these questionnaires came from existing paper versions. However, the layout is adjusted to fit the screen format and over the years some functionality has been implemented. The layout is standardized, but the standards for municipality and business questionnaires are different. The business questionnaires are now going to be moved from one web portal to another. Eventually the plan is also to move the municipality questionnaires to the same web portal as the business surveys.
In social surveys a few self administrative surveys have been run with a combination of paper and web questionnaires. However, computer assisted interviewing is the dominant data collection method and has until now seldom been combined with self completion web questionnaires. Now this is about to change too. A combination of web questionnaires and interviewing is planned for several of our main social surveys.
Evaluations of the quality of business survey questionnaires have uncovered great potentials for improvements. In addition, there is a need for harmonizing the design of municipality and business questionnaires. In the development of social survey web questionnaires we can learn from the experiences with business and municipality questionnaires.
We have a rather long tradition with cognitive interviewing and testing with an eyetracker during the development of new questionnaires. The main approach has been to carry out a few interviews in two or three iterations. Eyetracking and behavioural data have only been used to enrich the results from the cognitive interviews. This approach has worked well in the sense that it effectively reveals problems that the designer needs to address, but with the challenges we presently face we feel a need for a more scientific approach and an approach that realize more of the potentials of our testing instruments.
In self completion questionnaires the message is communicated by text combined with visual elements and interactive features. In his book on Internet and mail surveys, Don Dillman and his colleagues advocates a holistic approach where the different text, visual and functional elements convey a common meaning. Traditionally, however, our evaluation and testing methods have predominantly focused on question wordings and on one question element at a time. In order to meet the challenges we presently face we feel that we need tools that map how cognitive, visual and functional question elements play together in a better way than before. Moreover we think that we need to introduce a stronger quantitative element in our testing procedures than what has been common until now. At the same time as these challenges call for more extensive testing, we need also to take into consideration what is feasible in a production oriented organisation like ours.
In our presentation we will invite to a discussion about this challenges and thoughts for the future.